Add the LambdaTest gateway to incident-scribe and a one-command samples setup by amankumarslt · Pull Request #72 · LambdaTest/rook

amankumarslt · 2026-10-29T11:55:30Z

What

1. A lambdatest provider for incident-scribe (incident_scribe/llm.py)

Calls an OpenAI-compatible chat-completions gateway at $GATEWAY_URL/v1/chat/completions with HTTP Basic auth built from LT_USERNAME and LT_ACCESS_KEY. LT_MODEL is optional and defaults to auto.
Select it with LLM_PROVIDER=lambdatest (alias lt). It is also auto-detected, at the lowest priority, when all three variables are set and no OPENAI_API_KEY is.
Requests are non-streaming.
GATEWAY_URL must be https:// because the access key is sent with every call.

2. scripts/check_llm.py
Checks that the keys are present, the endpoint is up (unauthenticated GET /healthz) and the credentials are accepted (one 64-token call). It prints only variable names, never values.
Exit codes: 0 ok, 1 not configured, 2 credentials refused (401/403), 3 unreachable or request rejected. --config-only skips the network.

3. samples/setup.sh and samples/setup.ps1
One command sets up every sample, with no flags needed:

Checks Python 3.9+ and Node (22.13+ for the two node:sqlite samples).
Asks for LT_USERNAME and LT_ACCESS_KEY when they are empty (the key is typed hidden), and ends by printing . scripts/env.sh / . .\scripts\env.ps1, which load .env into your shell.
Creates incident-scribe/.env from .env.example (mode 600) and generates SCRIBE_API_TOKEN through scripts/ensure_token.py, which also repairs an empty, quoted or commented-out token line.
Installs the semantic-search embeddings, runs check_llm.py, then runs preflight.
Runs npm install and npm run setup for industry-agents.
Safe to re-run: an existing .env is kept.
Options: --no-semantic (-NoSemantic on Windows) and an optional list of sample names.

4. Docs and config

.env.example now defaults to the LambdaTest gateway (GATEWAY_URL=https://rook-samples.lambdatest.com).
Updated the provider tables in the incident-scribe README and docs/invoking.md, and added the setup command to samples/README.md.

Review fixes

Addresses the 15 inline review comments, with the failing tests committed first (7 failed before the fix).

check_llm.py no longer exits 1 ("not configured") on an unexpected answer. ModelError carries the HTTP status: no status is exit 3, 401/403 is exit 2, any other status or an error body is exit 3. A bad GATEWAY_URL is exit 1.
The health probe keeps the path in GATEWAY_URL.
Auto-detect no longer takes over from an OpenAI chat user.
Setup passes its chosen Python to preflight, reports a failing npm run setup, and prints a hint that fits the provider already in .env.

Behaviour changes to check

Embeddings now install by default in setup.sh. That is about a 600 MB download; use --no-semantic to skip it.
Each setup run makes one real 64-token model call, to prove the credentials work.

Testing

New tests for Basic auth, URL and default model; the missing-variable error; the status-based exit codes; the https guard; auto-detect precedence; and token repair. 18 of 19 pass.
The full setup was run from / on a fresh copy against a local fake gateway: exit 0, token generated, embeddings installed, checks and preflight passed.
check_llm.py was run against the fake gateway for good credentials, a wrong key (exit 2), a 404 (exit 3), unreachable (exit 3), a missing key (exit 1) and --config-only.
The real https://rook-samples.lambdatest.com/healthz returns 200 {"ok":true}.

Not verified

setup.ps1 has not been run. There is no PowerShell on the machine it was written on. It mirrors setup.sh, so please try it on Windows.
Real credentials have not been used against the real gateway; only /healthz was checked live.
test_incident_scribe.py::test_index_incident_flywheel fails on main too (it expects semantic mode and gets lexical-only, which needs embeddings installed). It is unrelated to this change.
The agent caps output at 1200, 1500 and 2500 tokens. A reasoning model behind auto could use up that budget before it writes an answer.
The incident-scribe README's ./scripts/preflight.sh line fails with "Permission denied", because the file is mode 644 in git. setup.sh calls it with bash. That line is left as it was.

🤖 Generated with Claude Code

…es setup incident-scribe gains a `lambdatest` provider: an OpenAI-compatible chat-completions gateway called with HTTP Basic auth built from LT_USERNAME and LT_ACCESS_KEY, at GATEWAY_URL/v1/chat/completions. The other providers are unchanged, and it is auto-detected last. scripts/check_llm.py reports whether the keys are present, the endpoint is up (GET /healthz) and the credentials are accepted, without printing secrets. Exit codes: 0 ok, 1 not configured, 2 refused, 3 unreachable. samples/setup.sh and setup.ps1 set up every sample with one command: prerequisites, .env files, the incident-scribe token, embeddings and the model check. .env.example now defaults to the LambdaTest gateway. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

samyakLambda

15 inline comments on 5f37d7f. The ones I'd fix before merge:

check_llm.py exits with code 1 ("not configured") on uncaught errors, so setup gives the wrong hint and still exits 0.
About 1 in 64 generated tokens starts with -, and argparse then rejects --token.
One failing npm run setup aborts setup.sh under pipefail, with no summary.
LambdaTest auto-detection outranks the chat fallback.

.env permissions and the cleartext-http credentials are small security fixes. The rest are edge cases and Windows-specific issues.

samyakLambda · 2026-10-29T12:27:52Z

+    parts = urllib.parse.urlsplit(endpoint)
+    url = f"{parts.scheme}://{parts.netloc}/healthz"
+    try:
+        with urllib.request.urlopen(url, timeout=15) as response:


Only ModelError, URLError, TimeoutError and OSError are caught, so any other exception ends in a traceback with Python's default exit code 1. Exit 1 is the "not configured" code, so setup prints the fill-in-your-keys hint, doesn't mark the sample failed, and still ends with "done." and exit 0.

Two ways to hit it:

GATEWAY_URL=rook-samples.lambdatest.com (no scheme): urlopen(':///healthz') raises ValueError: unknown url type.

A GATEWAY_URL that answers 200 with an HTML page: json.load in llm._request raises JSONDecodeError out of llm.call.

A final except Exception that prints the error and exits 3 (or a code of its own) would keep these out of the "not configured" path.

Mostly fixed in ba9646a. The new except Exception in main() covers the model request, but gateway_health() runs before it, and this urlopen still has only the HTTPError and URLError/TimeoutError/OSError handlers. Two GATEWAY_URL typos that pass the https check still escape (tested with plain urllib.request.urlopen):

https://rook-samples..lambdatest.com raises UnicodeEncodeError (idna: label empty)

a path with a space, e.g. https://host/llm gateway, raises http.client.InvalidURL

Both still end in a traceback with exit 1, so setup shows the "complete the provider settings" hint, doesn't mark incident-scribe failed, and finishes "done.". test_check_llm_never_reports_a_crash_as_not_configured stubs gateway_health, so it doesn't catch this. An except Exception as error: bad(...) after the two handlers, falling through to return False, would close it.

samyakLambda · 2026-10-29T12:27:52Z

+
+  if grep -Eq '^SCRIBE_API_TOKEN=[[:space:]]*$' .env; then
+    local token
+    token="$($PY -c 'import secrets; print(secrets.token_urlsafe(24))')"


secrets.token_urlsafe(24) starts with - about 1 time in 64 (1.56% over 200k samples). With such a token, the documented --token "$SCRIBE_API_TOKEN" form fails in argparse with argument --token: expected one argument, so every command fails. preflight.sh doesn't catch it, because it only tries a hard-coded wrong token. secrets.token_hex(24) avoids the problem. Same in setup.ps1 line 71.

samyakLambda · 2026-10-29T12:27:52Z

+  "$PY" scripts/check_llm.py || status=$?
+  case "$status" in
+    0) bash scripts/preflight.sh || FAILED="$FAILED incident-scribe" ;;
+    1)


This hint is the same whatever provider .env actually configures. Setup keeps an existing .env, so someone with a .env from the old template (LLM_PROVIDER=glm, empty GLM_API_KEY) is told to fill in LT_USERNAME/LT_ACCESS_KEY and that GATEWAY_URL is already set. Adding those keys changes nothing, because LLM_PROVIDER=glm still wins, and every re-run repeats the same advice. Printing the ModelError text from check_llm.py, or the resolved provider, would point to the real gap. setup.ps1 has the same hint.

samyakLambda · 2026-10-29T12:27:52Z

+  else
+    red "npm install failed"; FAILED="$FAILED industry-agents"; cd ..; return
+  fi
+  npm run setup --silent | while IFS= read -r line; do info "$line"; done


Under set -euo pipefail, a failing npm run setup fails this pipeline and the script exits right here: no FAILED entry, no "needs attention" summary, and the remaining samples are skipped. If setup-env.mjs throws (a missing .env.example, EACCES), refund-desk, triage-service, knowledge-vault and trip-weather-station never run.

setup.ps1 line 117 has the opposite problem: the failure is dropped and the run ends "done." with exit 0.

Wrapping it, e.g. if ! npm run setup --silent | while IFS= read -r line; do info "$line"; done; then FAILED="$FAILED industry-agents"; fi, would record the failure and carry on.

samyakLambda · 2026-10-29T12:27:52Z

            requested = "gemini"
        elif os.environ.get("OPENAI_ENDPOINT", "").strip():
            requested = "responses"
+        elif all(os.environ.get(name, "").strip() for name in ("GATEWAY_URL", "LT_USERNAME", "LT_ACCESS_KEY")):


The description says LambdaTest is the lowest-priority auto-detect, but this branch comes before the chat fallback. Take a Chat Completions user with OPENAI_API_KEY/OPENAI_CHAT_MODEL, no LLM_PROVIDER line, the GATEWAY_URL line from the new template, and LT_USERNAME/LT_ACCESS_KEY exported for Rook (common in CI). That user used to resolve to chat and now goes to rook-samples.lambdatest.com with their account key, without being told.

Checking for the chat keys first, or auto-selecting lambdatest only when no OpenAI key is set, would match the description. docs/invoking.md lines 88–90 still list the old order, and a precedence test would pin the behaviour.

samyakLambda · 2026-10-29T12:27:53Z

+
+  cd incident-scribe
+  if [ ! -f .env ]; then
+    cp .env.example .env


cp creates .env with the default umask (usually 0644). This file holds SCRIBE_API_TOKEN and, once filled in, LT_ACCESS_KEY, which signs in to the whole LambdaTest account. industry-agents' setupDemoEnv in shared/config.mjs writes its .env with mode 0o600 for this reason. install -m 600 .env.example .env, or umask 077 before the copy, would match it.

samyakLambda · 2026-10-29T12:27:53Z

+        $envText = Get-Content .env -Raw
+        if ($envText -match '(?m)^SCRIBE_API_TOKEN=[ \t]*\r?$') {
+            $token = (& $python -c "import secrets; print(secrets.token_urlsafe(24))").Trim()
+            $envText = [regex]::Replace($envText, '(?m)^SCRIBE_API_TOKEN=[ \t]*(?=\r?$)', "SCRIBE_API_TOKEN=$token", 1)


The static [regex]::Replace has no count overload, so the fourth argument binds to RegexOptions and 1 becomes IgnoreCase. Every matching line is replaced, case-insensitively, instead of only the first as in setup.sh. The instance method does take a count:

$envText = ([regex]'(?m)^SCRIBE_API_TOKEN=[ \t]*(?=\r?$)').Replace($envText, "SCRIBE_API_TOKEN=$token", 1)

samyakLambda · 2026-10-29T12:27:53Z


-def _call_chat(items: list[dict[str, Any]], tools: list[dict[str, Any]] | None, endpoint: str, key: str, model: str, max_tokens: int) -> dict[str, Any]:
+def _call_chat(items: list[dict[str, Any]], tools: list[dict[str, Any]] | None, endpoint: str, key: str, model: str, max_tokens: int, scheme: str = "Bearer") -> dict[str, Any]:
    body: dict[str, Any] = {"model": model, "messages": _chat_items(items), "max_tokens": max_tokens}


The lambdatest provider goes through _call_chat, which always sends max_tokens. If LT_MODEL=auto routes to an OpenAI reasoning model (o-series or gpt-5 class) and the gateway passes the body through unchanged, every call fails with a 400 Unsupported parameter: 'max_tokens', because those models expect max_completion_tokens. If the gateway does translate the field, reasoning can use up the 64/1200/1500/2500-token caps and leave the content empty. This lines up with the token-budget item the description lists as unverified, so it's worth confirming what auto resolves to.

Still open on ba9646a: the lambdatest provider still goes through _call_chat with max_tokens. Whether this is a real problem depends on what LT_MODEL=auto routes to and whether the gateway translates the field. If the gateway owner confirms either one, this can be closed as not applicable.

samyakLambda · 2026-10-29T12:27:53Z

+            Ok "kept existing .env"
+        }
+
+        $envText = Get-Content .env -Raw


On Windows PowerShell 5.1, Get-Content -Raw without -Encoding reads BOM-less UTF-8 in the ANSI code page, and WriteAllText writes it back as UTF-8, so non-ASCII text is double-encoded. On the first run, line 1 of .env.example ("See README.md → Configuration") becomes README.md â†’ Configuration, and any non-ASCII value in a kept .env is damaged the same way. Get-Content .env -Raw -Encoding UTF8 or [IO.File]::ReadAllText fixes it.

samyakLambda · 2026-10-29T12:27:53Z

+        ) if not value]
+        if missing:
+            raise ModelError("the LambdaTest gateway is not configured: " + ", ".join(missing))
+        return "lambdatest", _chat_url(endpoint), base64.b64encode(f"{username}:{access_key}".encode()).decode(), model


These Basic credentials are the LambdaTest account username and access key, and they go to whatever GATEWAY_URL holds. With an http:// URL (a typo, or a local proxy), the account-wide key is sent in cleartext on every call. Refusing a non-https GATEWAY_URL unless the host is localhost would close that.

Auto-detect precedence, cleartext credentials, HTTP status on ModelError, token generation edge cases, check_llm exit codes and the health probe path. These fail on the previous commit and are fixed in the next one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Check_llm no longer lets an unexpected answer exit 1 (the "not configured" code): errors carry the HTTP status, a bad GATEWAY_URL is caught as configuration, and any other failure exits 3. The gateway must be https (or localhost) because the access key goes with every call, the health probe keeps the URL's path, and auto-detect no longer takes over from an OpenAI chat user. Setup now repairs the token through scripts/ensure_token.py, which follows the loader's first-assignment-wins rule, creates .env as 0600, avoids tokens that start with a dash, passes the chosen Python to preflight, reports a failing `npm run setup`, checks Node 22.13 for the node:sqlite samples, and gives a hint that fits the provider in .env. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

samyakLambda

Follow-up on ba9646a: d3815c4 and ba9646a resolve 14 of the 15 earlier comments. Still open:

gateway_health() can still crash with exit 1 on two GATEWAY_URL typos (reply on the earlier exit-code thread).
max_tokens with reasoning models is untouched (reply on that thread).
CI doesn't run the incident-scribe tests. .github/workflows/test-samples.yml only has triage-service and industry-agents jobs, so the seven new tests in test_incident_scribe.py run only by hand, and the setup.sh/setup.ps1 changes have no tests. A pytest job for samples/incident-scribe would keep them running.
A docs nit, inline.

Reviewed by reading the diff; I didn't run the test suite locally.

samyakLambda · 2026-10-29T13:10:05Z

-Responses, or Chat Completions from the variables that are present, in that
-order.
+Responses, LambdaTest (when `GATEWAY_URL`, `LT_USERNAME` and `LT_ACCESS_KEY` are
+all set and no `OPENAI_API_KEY` is), or Chat Completions from the variables that


Nit: the code also skips LambdaTest when OPENAI_INDIA_API_KEY is set (llm.py line 40), so this could say "all set and neither OPENAI_API_KEY nor OPENAI_INDIA_API_KEY is".

scripts/env.sh and env.ps1 load .env into the current shell (source them), with the desk's rules: first assignment wins, the shell wins over the file. setup.sh and setup.ps1 now ask for LT_USERNAME and LT_ACCESS_KEY when they are empty, with the key typed hidden, and print the env line to run next. The docs now cover the loader and the https requirement. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

amankumarslt requested review from gauravBansal06 and vipulkv as code owners October 29, 2026 11:55

samyakLambda reviewed Oct 29, 2026

View reviewed changes

amankumarslt and others added 2 commits October 29, 2026 18:03

samyakLambda reviewed Oct 29, 2026

View reviewed changes

samyakLambda merged commit e1c54df into main Oct 29, 2026
8 checks passed

samyakLambda deleted the feat/incident-scribe-lambdatest-gateway-and-samples-setup branch October 29, 2026 13:18

Conversation

amankumarslt commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Review fixes

Behaviour changes to check

Testing

Not verified

Uh oh!

samyakLambda left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samyakLambda left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

amankumarslt commented Oct 29, 2026 •

edited

Loading